See here for AOP spectra from PRLA buoy location
This notebook summarizes cholorphyll, DOM, and suspended matter data from sampling and sensors. These are inputs to forward model or reponse for statistical models.
Sampling data is chl a, pheophytin, DOC, TOC, UV abs 254 and 280, Fe, TSS, TDS
Data for sample collected closest in time to each flight date is reported, usually one value for each flight year
Sensor data is chl, fDOM, turbidity, uv abs 254 (from SUNA), uv abs 350 (from SUNA)
Closet in time sensor values are from a moving average of sensor time series
Sensor data at time of ground based sample collection is compared for chl, fDOM vs DOC, fDOM vs uv254, TSS vs turbidity, uv 254
L0 SUNA data needs to be downloaded and processed separately in processing project
Updating the AOS and AIS data is handled separately in neon-aqu-aos.
AOP flight times are determined from L1 radiance metadata.

Summary of data overlap for PRLA

Green flightlines from multiple dates in 2019 (July 26, 27) and 2020 (June 24, July 2)
In 2019, chl sampling from July 11 (2 weeks before), chemistry August 6 (11 days after), and sensor data is available
In 2020, chl sampling from June 15 (9 days before), chemistry data not all available from closest sampling date (only TOC not DOC). sensor data available for first flight date but not later date (which had a higher spectra with more pronounced peaks). The chl sample from the next bout was > 100 ug/L
Also AOP data collected in 2016 and 2017 under yellow conditions but no ground-based data

mysite = 'PRLA'

I. Chlorophyll

I.(A) AOS Sampling

From processed chl a files on Box
reported in ug/L
seasonal sampling, 2 analytes

See how many locations chl data is collected from

See how many locations chl data is collected from (variable Y axis)

Availability of chlorophyll AOS data for each flight date

Default threshold 1 = 30 days
Default threshold 2 = 60 days

flightdate	aos_before	aos_after	days_before	days_after	min_days	meets_thresh1	meets_thresh2
2016-06-29	NA	2019-07-11	NA	1107	1107	FALSE	FALSE
2017-06-21	NA	2019-07-11	NA	750	750	FALSE	FALSE
2017-06-26	NA	2019-07-11	NA	745	745	FALSE	FALSE
2019-07-26	2019-07-11	2019-09-10	15	46	15	TRUE	TRUE
2019-07-27	2019-07-11	2019-09-10	16	45	16	TRUE	TRUE
2019-07-30	2019-07-11	2019-09-10	19	42	19	TRUE	TRUE
2020-06-24	2020-06-15	2020-09-29	9	97	9	TRUE	TRUE
2020-06-26	2020-06-15	2020-09-29	11	95	11	TRUE	TRUE
2020-07-02	2020-06-15	2020-09-29	17	89	17	TRUE	TRUE

flightdate	aos_match	days
2019-07-26	2019-07-11	15
2019-07-27	2019-07-11	16
2019-07-30	2019-07-11	19
2020-06-24	2020-06-15	9
2020-06-26	2020-06-15	11
2020-07-02	2020-06-15	17

Make sure plant algae lab units are all micrograms per liter
Check sample conditions

## [1] "microgramsPerLiter"

## [1] "condition ok"

chl AOS x flightdates

These are the closest chl AOS values for each flight date

aos_match	flightdate	days	pheophytin	chlorophyll a
2019-07-11	2019-07-26	15	3.01	2.31
2019-07-11	2019-07-27	16	3.01	2.31
2019-07-11	2019-07-30	19	3.01	2.31
2020-06-15	2020-06-24	9	4.20	3.45
2020-06-15	2020-06-26	11	4.20	3.45
2020-06-15	2020-07-02	17	4.20	3.45

Algal Community

Algal types include:

Bacillariophyceae
Cyanobacteria
Chlorophyta and Charophyta
Xanthophyceae
Unclassified

Algal divisions include:

Chrysophyta
Cyanophyta
Chlorophyta

Algal classes include:

Bacillariophyceae
Myxophyceae
Chlorophyceae
Chrysophyceae

Abundance is reported in a range of units:

number of cells, natural units, or valves (Chrysophyta/Bacillariophyceae)
cell density
biovolume density

Starting in 2018 algal type no longer reported?

## 
## ── Column specification ────────────────────────────────────────────────────────
## cols(
##   namedLocation = col_character(),
##   collectDate = col_date(format = ""),
##   sampleID = col_character(),
##   algalParameter = col_character(),
##   class = col_character(),
##   sum_param = col_double(),
##   rel_abun = col_double()
## )

## 
## ── Column specification ────────────────────────────────────────────────────────
## cols(
##   namedLocation = col_character(),
##   collectDate = col_date(format = ""),
##   sampleID = col_character(),
##   algalParameter = col_character(),
##   division = col_character(),
##   sum_param = col_double(),
##   rel_abun = col_double()
## )

## 
## ── Column specification ────────────────────────────────────────────────────────
## cols(
##   namedLocation = col_character(),
##   collectDate = col_date(format = ""),
##   sampleID = col_character(),
##   algalParameter = col_character(),
##   algalType = col_character(),
##   sum_param = col_double(),
##   rel_abun = col_double()
## )

## Warning: Removed 1 rows containing missing values (position_stack).

## Warning: Removed 1 rows containing missing values (position_stack).

## Warning: Removed 1 rows containing missing values (position_stack).

I.(B) Chl AIS time series

The EXO total algae sensor is a dual‐channel fluorometer that uses a 470nm excitation beam that excites chlorophyll a and a second 590 nm excitation beam that excites the phyocyanin accessory pigment found in blue‐green algae (cyanobacteria). Chlorophyll concentration is a biogeochemically relavant parameter that is readily available by remote sensing and can be can serve as a proxy for phytoplankton biomass and light attenuation (Oestreich et al., 2016, Ganju et al., 2014, Jaud et al., 2012)

Sensor chl data availability

flightline_datetime	check_30day	check_10day	check_1day	check_12hr
2016-06-29 15:45:47	FALSE	FALSE	FALSE	FALSE
2016-06-29 15:52:09	FALSE	FALSE	FALSE	FALSE
2016-06-29 17:22:24	FALSE	FALSE	FALSE	FALSE
2017-06-21 20:58:28	FALSE	FALSE	FALSE	FALSE
2017-06-26 15:33:21	FALSE	FALSE	FALSE	FALSE
2017-06-26 16:04:51	FALSE	FALSE	FALSE	FALSE
2017-06-26 16:11:11	FALSE	FALSE	FALSE	FALSE
2019-07-26 15:20:23	TRUE	TRUE	TRUE	TRUE
2019-07-26 16:22:11	TRUE	TRUE	TRUE	TRUE
2019-07-26 22:09:25	TRUE	TRUE	TRUE	TRUE
2019-07-27 15:27:04	TRUE	TRUE	TRUE	TRUE
2019-07-30 15:34:20	TRUE	TRUE	TRUE	TRUE
2020-06-24 17:36:27	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:26:15	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:18:44	TRUE	TRUE	TRUE	TRUE
2020-06-26 19:28:06	TRUE	TRUE	TRUE	FALSE
2020-07-02 17:14:10	TRUE	TRUE	FALSE	FALSE

2019 chl AIS plots

Filtered to only 0 quality flags

2020 chl AIS plots

No QF filters on these plots
Is QF because of lack of calibration?

2019 Chl avg

Smooth the chl time series using a rolling mean with zoo::rollapply.
First convert to a regular 5 minute time series using tsibble

Get the moving average value closet to flight time.

5min is the original data from that row
ma01 = 1 hour window, ma03 = 3 hour window, and so on
chl_ma04u is mean, the others are all mean

collectDateTime	datetime	chl5min	chl_ma01	chl_ma03	chl_ma04	chl_ma04u	chl_ma06	chl_ma12
2019-07-26 15:20:23	2019-07-26 15:20:00	5.83	6.245	6.285	6.290	6.312174	6.350	6.610
2019-07-26 16:22:11	2019-07-26 16:20:00	6.70	6.175	6.285	6.310	6.327500	6.425	6.610
2019-07-26 22:09:25	2019-07-26 22:10:00	7.47	7.665	7.735	7.695	7.881042	7.660	7.650
2019-07-27 15:27:04	2019-07-27 15:25:00	8.16	8.195	8.380	8.400	8.477083	8.455	8.655
2019-07-30 15:34:20	2019-07-30 15:35:00	18.12	17.850	17.460	17.460	17.539130	17.860	19.670

2020 Chl avg

Smooth the chl time series using a rolling mean with zoo::rollapply.
First convert to a regular 5 minute time series using tsibble
Using median for 2020 because of QA issues? (or could remove outliers before calculating running median)

Get the moving average value closet to flight time

collectDateTime	datetime	chl5min	chl_ma01	chl_ma03	chl_ma04	chl_ma04u	chl_ma06	chl_ma12
2020-06-24 17:36:27	2020-06-24 17:35:00	2.54	2.320	2.48	2.545	2.638333	2.715	3.150
2020-06-24 16:26:15	2020-06-24 16:25:00	2.74	2.565	2.56	2.565	2.581875	2.680	3.165
2020-06-24 16:18:44	2020-06-24 16:20:00	2.54	2.565	2.56	2.585	2.587917	2.680	3.175
2020-06-26 19:28:06	2020-06-26 19:30:00	NA	NA	NA	NA	NaN	NA	NA
2020-07-02 17:14:10	2020-07-02 17:15:00	NA	NA	NA	NA	NaN	NA	NA

I.(C) Chl AIS vs AOS

use chla_df date and times with the aop/ais matching function to determine which sampling dates have sensor data from same day around time of sampling.

already sort of done for suna data… follow this pattern

Get chl sensor data from all AOS sampling dates (use ais with buffer 2 fucntion?)
show what is available
calculate rolling averages of chl time series
get rolling average values for time closest to sampling using which min on difftime
little overlap because of seasonal removal of buoys

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2019-07-11 13:53:00	TRUE	TRUE	TRUE	TRUE
2019-09-10 13:37:00	TRUE	TRUE	TRUE	TRUE
2020-06-15 15:09:00	TRUE	TRUE	TRUE	TRUE
2020-09-29 13:21:00	FALSE	FALSE	FALSE	FALSE
2021-04-20 14:40:00	FALSE	FALSE	FALSE	FALSE

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2019-07-11 13:53:00	TRUE	TRUE	TRUE	TRUE
2019-09-10 13:37:00	TRUE	TRUE	TRUE	TRUE
2020-06-15 15:09:00	TRUE	TRUE	TRUE	TRUE

collect_date	pheophytin	chla	chl_tot
2019-07-11	3.01	2.31	5.32
2019-09-10	3.08	2.79	5.87
2020-06-15	4.20	3.45	7.65
2020-09-29	72.40	124.44	196.84
2021-04-20	21.38	20.03	41.41

Compare AOS and AIS

II. DOM

II.(A) AOS Sampling

From processed Surface water chemistry files on Box

EPA Method description:

The UVA procedure requires that the sample be passed through a 0.45 um filter and transferred to quartz cell. It is then placed in a spectrophotometer to measure the UV absorbance at 254 nm and reported in cm -1.
The SUVA procedure requires both the DOC and UVA measurement. The SUVA is then calculated by dividing the UV absorbance of the sample (in cm -1) by the DOC of the sample (in mg/L) and then multiplying by 100 cm/M. SUVA is reported in units of L/mg-M. The formula for the SUVA may be found in Section 12.2.

Ignoring data before 2016

## Warning: Removed 2 rows containing missing values (geom_point).

Correct older UV Absorbance wavelengths from 250 to 254, assuming all 250 nm should be 254.

swchem_site_df <- swchem_site_df %>%
  dplyr::mutate(analyte = dplyr::case_when(analyte == 'UV Absorbance (250 nm)' ~
                                             'UV Absorbance (254 nm)', TRUE ~ analyte))

## Warning: Removed 2 rows containing missing values (geom_point).

How much does SUVA 254/280 ratio change?

How much does organic carbon dissolved proportion change?

Adjusts sample IDs in a new column to group together raw and filtered, while keeping track of duplicates

## Warning: Removed 11 rows containing missing values (geom_point).

Match DOM AOS and AOP data

Need to do this separately for each analyte because they arent always all reported.

DOC x flightdates

These are the closest DOC values (mg/L) for each flight date

A list of values means there are duplicate samples (multiple sampleIDs) from same date.

flightdate	aos_before	aos_after	days_before	days_after	min_days	meets_thresh1	meets_thresh2
2016-06-29	2014-06-19	2016-10-19	741	112	112	FALSE	FALSE
2017-06-21	2017-06-08	2017-07-06	13	15	13	TRUE	TRUE
2017-06-26	2017-06-08	2017-07-06	18	10	10	TRUE	TRUE
2019-07-26	2019-07-01	2019-08-06	25	11	11	TRUE	TRUE
2019-07-27	2019-07-01	2019-08-06	26	10	10	TRUE	TRUE
2019-07-30	2019-07-01	2019-08-06	29	7	7	TRUE	TRUE
2020-06-24	2020-06-09	2020-08-04	15	41	15	TRUE	TRUE
2020-06-26	2020-06-09	2020-08-04	17	39	17	TRUE	TRUE
2020-07-02	2020-06-09	2020-08-04	23	33	23	TRUE	TRUE

flightdate	aos_match	days
2017-06-21	2017-06-08	13
2017-06-26	2017-07-06	10
2019-07-26	2019-08-06	11
2019-07-27	2019-08-06	10
2019-07-30	2019-08-06	7
2020-06-24	2020-06-09	15
2020-06-26	2020-06-09	17
2020-07-02	2020-06-09	23

aos_match	flightdate	days	DOC
2017-06-08	2017-06-21	13	29.97
2017-07-06	2017-06-26	10	31.68, 31.81, 32.11
2019-08-06	2019-07-26	11	35.01
2019-08-06	2019-07-27	10	35.01
2019-08-06	2019-07-30	7	35.01
2020-06-09	2020-06-24	15	26.72
2020-06-09	2020-06-26	17	26.72
2020-06-09	2020-07-02	23	26.72

UV abs x flightdates

These are the closest UV abs values (per 1 cm) for each flight date

A list of values means there are duplicate samples (multiple sampleIDs) from same date.

flightdate	aos_before	aos_after	days_before	days_after	min_days	meets_thresh1	meets_thresh2
2016-06-29	NA	2016-10-19	NA	112	112	FALSE	FALSE
2017-06-21	2017-06-08	2017-07-06	13	15	13	TRUE	TRUE
2017-06-26	2017-06-08	2017-07-06	18	10	10	TRUE	TRUE
2019-07-26	2019-07-01	2019-08-06	25	11	11	TRUE	TRUE
2019-07-27	2019-07-01	2019-08-06	26	10	10	TRUE	TRUE
2019-07-30	2019-07-01	2019-08-06	29	7	7	TRUE	TRUE
2020-06-24	2019-12-04	2020-08-04	203	41	41	FALSE	TRUE
2020-06-26	2019-12-04	2020-08-04	205	39	39	FALSE	TRUE
2020-07-02	2019-12-04	2020-08-04	211	33	33	FALSE	TRUE

flightdate	aos_match	days
2017-06-21	2017-06-08	13
2017-06-26	2017-07-06	10
2019-07-26	2019-08-06	11
2019-07-27	2019-08-06	10
2019-07-30	2019-08-06	7
2020-06-24	2020-08-04	41
2020-06-26	2020-08-04	39
2020-07-02	2020-08-04	33

aos_match	flightdate	days	UV Absorbance (254 nm)
2017-06-08	2017-06-21	13	0.5578
2017-07-06	2017-06-26	10	0.5566, 0.5710, 0.5653
2019-08-06	2019-07-26	11	0.7538
2019-08-06	2019-07-27	10	0.7538
2019-08-06	2019-07-30	7	0.7538
2020-08-04	2020-06-24	41	0.5646
2020-08-04	2020-06-26	39	0.5646
2020-08-04	2020-07-02	33	0.5646

flightdate	aos_before	aos_after	days_before	days_after	min_days	meets_thresh1	meets_thresh2
2016-06-29	NA	2016-10-19	NA	112	112	FALSE	FALSE
2017-06-21	2017-06-08	2017-07-06	13	15	13	TRUE	TRUE
2017-06-26	2017-06-08	2017-07-06	18	10	10	TRUE	TRUE
2019-07-26	2019-07-01	2019-08-06	25	11	11	TRUE	TRUE
2019-07-27	2019-07-01	2019-08-06	26	10	10	TRUE	TRUE
2019-07-30	2019-07-01	2019-08-06	29	7	7	TRUE	TRUE
2020-06-24	2019-12-04	2020-08-04	203	41	41	FALSE	TRUE
2020-06-26	2019-12-04	2020-08-04	205	39	39	FALSE	TRUE
2020-07-02	2019-12-04	2020-08-04	211	33	33	FALSE	TRUE

flightdate	aos_match	days
2017-06-21	2017-06-08	13
2017-06-26	2017-07-06	10
2019-07-26	2019-08-06	11
2019-07-27	2019-08-06	10
2019-07-30	2019-08-06	7
2020-06-24	2020-08-04	41
2020-06-26	2020-08-04	39
2020-07-02	2020-08-04	33

aos_match	flightdate	days	UV Absorbance (280 nm)
2017-06-08	2017-06-21	13	0.2876
2017-07-06	2017-06-26	10	0.2887, 0.2961, 0.2950
2019-08-06	2019-07-26	11	0.3852
2019-08-06	2019-07-27	10	0.3852
2019-08-06	2019-07-30	7	0.3852
2020-08-04	2020-06-24	41	0.3118
2020-08-04	2020-06-26	39	0.3118
2020-08-04	2020-07-02	33	0.3118

TOC x flightdates

TOC is reported in milligramsPerLiter

flightdate	aos_before	aos_after	days_before	days_after	min_days	meets_thresh1	meets_thresh2
2016-06-29	2014-06-19	2016-10-19	741	112	112	FALSE	FALSE
2017-06-21	2017-06-08	2017-07-06	13	15	13	TRUE	TRUE
2017-06-26	2017-06-08	2017-07-06	18	10	10	TRUE	TRUE
2019-07-26	2019-07-01	2019-08-06	25	11	11	TRUE	TRUE
2019-07-27	2019-07-01	2019-08-06	26	10	10	TRUE	TRUE
2019-07-30	2019-07-01	2019-08-06	29	7	7	TRUE	TRUE
2020-06-24	2020-06-09	2020-06-30	15	6	6	TRUE	TRUE
2020-06-26	2020-06-09	2020-06-30	17	4	4	TRUE	TRUE
2020-07-02	2020-06-30	2020-08-04	2	33	2	TRUE	TRUE

flightdate	aos_match	days
2017-06-21	2017-06-08	13
2017-06-26	2017-07-06	10
2019-07-26	2019-08-06	11
2019-07-27	2019-08-06	10
2019-07-30	2019-08-06	7
2020-06-24	2020-06-30	6
2020-06-26	2020-06-30	4
2020-07-02	2020-06-30	2

aos_match	flightdate	days	TOC
2017-06-08	2017-06-21	13	34.12
2017-07-06	2017-06-26	10	37.87, 38.11, 38.82
2019-08-06	2019-07-26	11	36.05
2019-08-06	2019-07-27	10	36.05
2019-08-06	2019-07-30	7	36.05
2020-06-30	2020-06-24	6	26.05, 26.49, 26.18
2020-06-30	2020-06-26	4	26.05, 26.49, 26.18
2020-06-30	2020-07-02	2	26.05, 26.49, 26.18

Fe x flightdates

Fe is reported in milligramsPerLiter

flightdate	aos_before	aos_after	days_before	days_after	min_days	meets_thresh1	meets_thresh2
2016-06-29	2014-06-19	2016-10-19	741	112	112	FALSE	FALSE
2017-06-21	2017-06-08	2017-07-06	13	15	13	TRUE	TRUE
2017-06-26	2017-06-08	2017-07-06	18	10	10	TRUE	TRUE
2019-07-26	2019-07-01	2019-08-06	25	11	11	TRUE	TRUE
2019-07-27	2019-07-01	2019-08-06	26	10	10	TRUE	TRUE
2019-07-30	2019-07-01	2019-08-06	29	7	7	TRUE	TRUE
2020-06-24	2019-12-04	2021-05-04	203	314	203	FALSE	FALSE
2020-06-26	2019-12-04	2021-05-04	205	312	205	FALSE	FALSE
2020-07-02	2019-12-04	2021-05-04	211	306	211	FALSE	FALSE

flightdate	aos_match	days
2017-06-21	2017-06-08	13
2017-06-26	2017-07-06	10
2019-07-26	2019-08-06	11
2019-07-27	2019-08-06	10
2019-07-30	2019-08-06	7

aos_match	flightdate	days	Fe
2017-06-08	2017-06-21	13	5e-04
2017-07-06	2017-06-26	10	5e-04, 5e-04, 5e-04
2019-08-06	2019-07-26	11	0.01
2019-08-06	2019-07-27	10	0.01
2019-08-06	2019-07-30	7	0.01

II.(B) fDOM AIS time series

The EXO fDOM sensor is a fluorometer with a single excitation/emission pair (365nm/480nm) used to detect the fluorescent fraction of the chromophoric DOM when exposed to near‐UV light. Because of the impacts of temperature and water column absorbance (from a combination of dissolved and particulate compounds) on these readings corrections must be applied to the calibrated data.

availability of processed fDOM time series within different time windows

flightline_datetime	check_30day	check_10day	check_1day	check_12hr
2016-06-29 15:45:47	FALSE	FALSE	FALSE	FALSE
2016-06-29 15:52:09	FALSE	FALSE	FALSE	FALSE
2016-06-29 17:22:24	FALSE	FALSE	FALSE	FALSE
2017-06-21 20:58:28	FALSE	FALSE	FALSE	FALSE
2017-06-26 15:33:21	FALSE	FALSE	FALSE	FALSE
2017-06-26 16:04:51	FALSE	FALSE	FALSE	FALSE
2017-06-26 16:11:11	FALSE	FALSE	FALSE	FALSE
2019-07-26 15:20:23	TRUE	TRUE	TRUE	TRUE
2019-07-26 16:22:11	TRUE	TRUE	TRUE	TRUE
2019-07-26 22:09:25	TRUE	TRUE	TRUE	TRUE
2019-07-27 15:27:04	TRUE	TRUE	TRUE	TRUE
2019-07-30 15:34:20	TRUE	TRUE	TRUE	TRUE
2020-06-24 17:36:27	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:26:15	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:18:44	TRUE	TRUE	TRUE	TRUE
2020-06-26 19:28:06	TRUE	TRUE	TRUE	FALSE
2020-07-02 17:14:10	TRUE	TRUE	FALSE	FALSE

2019 fDOM plots

## Warning: Removed 3170 rows containing missing values (geom_point).

## Warning: Removed 21 rows containing missing values (geom_point).

## `geom_smooth()` using method = 'loess' and formula 'y ~ x'

## `geom_smooth()` using method = 'loess' and formula 'y ~ x'

## `geom_smooth()` using method = 'loess' and formula 'y ~ x'

2020 fDOM plots

Not filtered on QFs
QF because of lack of calibration?

## Warning: Removed 8658 rows containing missing values (geom_point).

## Warning: Removed 2552 rows containing missing values (geom_point).

## Warning: Removed 444 row(s) containing missing values (geom_path).

## Warning: Removed 444 rows containing missing values (geom_point).

## Warning: Removed 4737 rows containing missing values (geom_point).

2019 fDOM avg

Smooth the fdom time series using a rolling mean with zoo::rollapply.
First convert to a regular 5 minute time series using tsibble

Get the moving average value closet to flight time.

5min is the original data from that row
ma01 = 1 hour window, ma03 = 3 hour window, and so on
chl_ma04u is mean, the others are all mean

collectDateTime	datetime	fdom5min	ma01	ma03	ma04	ma04u	ma06	ma12
2019-07-26 15:20:23	2019-07-26 15:20:00	109.76	109.620	109.620	109.630	109.6151	109.630	109.63
2019-07-26 16:22:11	2019-07-26 16:20:00	109.22	109.260	109.260	109.260	109.2767	109.260	109.26
2019-07-26 22:09:25	2019-07-26 22:10:00	108.23	108.075	108.065	108.065	108.0837	108.070	108.09
2019-07-27 15:27:04	2019-07-27 15:25:00	105.65	105.760	105.745	105.745	105.7490	105.745	105.80
2019-07-30 15:34:20	2019-07-30 15:35:00	100.23	100.380	100.350	100.310	100.2890	100.340	100.35

2020 fDOM avg

Smooth the fdom time series using a rolling mean with zoo::rollapply.
First convert to a regular 5 minute time series using tsibble

Get the moving average value closet to flight time.

5min is the original data from that row
ma01 = 1 hour window, ma03 = 3 hour window, and so on
chl_ma04u is mean, the others are all mean

collectDateTime	datetime	fdom5min	ma01	ma03	ma04	ma04u	ma06	ma12
2020-06-24 17:36:27	2020-06-24 17:35:00	69.59	69.625	69.700	69.730	69.53479	69.865	70.200
2020-06-24 16:26:15	2020-06-24 16:25:00	70.02	69.855	69.880	69.880	69.94437	69.850	70.105
2020-06-24 16:18:44	2020-06-24 16:20:00	69.73	69.945	69.955	69.955	69.96188	69.850	70.105
2020-06-26 19:28:06	2020-06-26 19:30:00	NA	NA	NA	NA	NaN	NA	NA
2020-07-02 17:14:10	2020-07-02 17:15:00	NA	NA	NA	NA	NaN	NA	NA

II.(C) fDOM vs AOS DOM

use swchem_df date and times with the aop/ais matching function to determine which sampling dates have sensor data from same day around time of sampling.
Get fDOM sensor data from all AOS sampling dates (use ais with buffer2 function?)
show what is available
calculate rolling averages of fDOM time series
get rolling average values for time closest to sampling using which min on difftime
little overlap because of seasonal removal of buoys

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2014-05-14 15:15:00	FALSE	FALSE	FALSE	FALSE
2014-06-19 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-10-19 15:30:00	FALSE	FALSE	FALSE	FALSE
2016-11-16 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-12-19 17:53:00	FALSE	FALSE	FALSE	FALSE
2017-01-18 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-02-13 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-03-22 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-04-12 15:15:00	FALSE	FALSE	FALSE	FALSE
2017-05-03 14:20:00	FALSE	FALSE	FALSE	FALSE
2017-06-08 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-07-06 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-08-03 14:50:00	FALSE	FALSE	FALSE	FALSE
2017-09-06 14:15:00	FALSE	FALSE	FALSE	FALSE
2017-10-19 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-11-01 15:15:00	FALSE	FALSE	FALSE	FALSE
2018-01-04 17:00:00	FALSE	FALSE	FALSE	FALSE
2018-02-13 16:45:00	FALSE	FALSE	FALSE	FALSE
2018-03-13 16:05:00	FALSE	FALSE	FALSE	FALSE
2018-04-17 16:03:00	FALSE	FALSE	FALSE	FALSE
2018-05-01 16:26:00	FALSE	FALSE	FALSE	FALSE
2018-06-05 15:40:00	FALSE	FALSE	FALSE	FALSE
2018-07-05 15:20:00	TRUE	TRUE	TRUE	TRUE
2018-08-07 16:16:00	TRUE	TRUE	TRUE	TRUE
2018-09-04 15:50:00	TRUE	TRUE	TRUE	TRUE
2018-10-16 16:03:00	TRUE	TRUE	TRUE	TRUE
2018-10-30 16:07:00	FALSE	FALSE	FALSE	FALSE
2018-12-04 16:42:00	FALSE	FALSE	FALSE	FALSE
2019-01-10 17:10:00	FALSE	FALSE	FALSE	FALSE
2019-02-11 17:55:00	FALSE	FALSE	FALSE	FALSE
2019-03-26 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-05-07 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-06-04 15:25:00	TRUE	TRUE	TRUE	TRUE
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE
2019-12-04 17:40:00	FALSE	FALSE	FALSE	FALSE
2020-01-09 15:12:00	FALSE	FALSE	FALSE	FALSE
2020-02-11 15:50:00	FALSE	FALSE	FALSE	FALSE
2020-03-10 16:15:00	FALSE	FALSE	FALSE	FALSE
2020-06-09 15:35:00	TRUE	TRUE	TRUE	TRUE
2020-06-30 15:22:00	FALSE	FALSE	FALSE	FALSE
2020-08-04 15:07:00	FALSE	FALSE	FALSE	FALSE
2020-09-02 13:27:00	FALSE	FALSE	FALSE	FALSE
2021-01-07 16:08:00	FALSE	FALSE	FALSE	FALSE
2021-02-02 17:15:00	FALSE	FALSE	FALSE	FALSE
2021-03-02 18:05:00	FALSE	FALSE	FALSE	FALSE
2021-04-07 15:45:00	FALSE	FALSE	FALSE	FALSE
2021-05-04 15:22:00	FALSE	FALSE	FALSE	FALSE

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2018-07-05 15:20:00	TRUE	TRUE	TRUE	TRUE
2018-08-07 16:16:00	TRUE	TRUE	TRUE	TRUE
2018-09-04 15:50:00	TRUE	TRUE	TRUE	TRUE
2018-10-16 16:03:00	TRUE	TRUE	TRUE	TRUE
2019-06-04 15:25:00	TRUE	TRUE	TRUE	TRUE
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE
2020-06-09 15:35:00	TRUE	TRUE	TRUE	TRUE

collect_date	Fe	TOC	DOC	abs254	abs280
2018-07-05	0.0005	42.81	34.46	0.6168	0.3113
2018-08-07	0.0005	47.99	37.81	0.6288	0.3172
2018-09-04	0.0005	41.88	39.81	0.6569	0.3274
2018-10-16	0.0005	42.74	39.67	0.6455	0.3171
2019-06-04	0.0005	38.82	35.19	0.5498	0.2846
2019-07-01	0.0042	36.23	35.90	0.7863	0.4191
2019-07-01	0.0048	34.88	35.91	0.7881	0.4111
2019-07-01	0.0040	36.78	35.79	0.7881	0.4048
2019-08-06	0.0100	36.05	35.01	0.7538	0.3852
2019-09-04	0.0025	33.08	32.87	0.6865	0.3524
2019-09-04	0.0030	32.76	33.03	0.6967	0.3607
2019-09-04	0.0026	32.82	33.16	0.6503	0.3363
2019-10-16	0.0005	27.63	27.56	0.6416	0.3293
2020-06-09	NA	26.65	26.72	NA	NA

Same plot as above with only QF = 0 data

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2014-05-14 15:15:00	FALSE	FALSE	FALSE	FALSE
2014-06-19 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-10-19 15:30:00	FALSE	FALSE	FALSE	FALSE
2016-11-16 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-12-19 17:53:00	FALSE	FALSE	FALSE	FALSE
2017-01-18 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-02-13 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-03-22 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-04-12 15:15:00	FALSE	FALSE	FALSE	FALSE
2017-05-03 14:20:00	FALSE	FALSE	FALSE	FALSE
2017-06-08 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-07-06 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-08-03 14:50:00	FALSE	FALSE	FALSE	FALSE
2017-09-06 14:15:00	FALSE	FALSE	FALSE	FALSE
2017-10-19 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-11-01 15:15:00	FALSE	FALSE	FALSE	FALSE
2018-01-04 17:00:00	FALSE	FALSE	FALSE	FALSE
2018-02-13 16:45:00	FALSE	FALSE	FALSE	FALSE
2018-03-13 16:05:00	FALSE	FALSE	FALSE	FALSE
2018-04-17 16:03:00	FALSE	FALSE	FALSE	FALSE
2018-05-01 16:26:00	FALSE	FALSE	FALSE	FALSE
2018-06-05 15:40:00	FALSE	FALSE	FALSE	FALSE
2018-07-05 15:20:00	FALSE	FALSE	FALSE	FALSE
2018-08-07 16:16:00	FALSE	FALSE	FALSE	FALSE
2018-09-04 15:50:00	FALSE	FALSE	FALSE	FALSE
2018-10-16 16:03:00	FALSE	FALSE	FALSE	FALSE
2018-10-30 16:07:00	FALSE	FALSE	FALSE	FALSE
2018-12-04 16:42:00	FALSE	FALSE	FALSE	FALSE
2019-01-10 17:10:00	FALSE	FALSE	FALSE	FALSE
2019-02-11 17:55:00	FALSE	FALSE	FALSE	FALSE
2019-03-26 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-05-07 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-06-04 15:25:00	FALSE	FALSE	FALSE	FALSE
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE
2019-12-04 17:40:00	FALSE	FALSE	FALSE	FALSE
2020-01-09 15:12:00	FALSE	FALSE	FALSE	FALSE
2020-02-11 15:50:00	FALSE	FALSE	FALSE	FALSE
2020-03-10 16:15:00	FALSE	FALSE	FALSE	FALSE
2020-06-09 15:35:00	FALSE	FALSE	FALSE	FALSE
2020-06-30 15:22:00	FALSE	FALSE	FALSE	FALSE
2020-08-04 15:07:00	FALSE	FALSE	FALSE	FALSE
2020-09-02 13:27:00	FALSE	FALSE	FALSE	FALSE
2021-01-07 16:08:00	FALSE	FALSE	FALSE	FALSE
2021-02-02 17:15:00	FALSE	FALSE	FALSE	FALSE
2021-03-02 18:05:00	FALSE	FALSE	FALSE	FALSE
2021-04-07 15:45:00	FALSE	FALSE	FALSE	FALSE
2021-05-04 15:22:00	FALSE	FALSE	FALSE	FALSE

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE

## Warning: Removed 641 rows containing missing values (geom_point).

Compare AOS and AIS

Colors indicate month

II.(D) SUNA AIS time series

nitrate sensor 15 minute interval time series
value is from a burst of readings after lamp warms up
error bars on these plots are std error of mean, not expanded uncertainty like for other sensors

flightline_datetime	check_30day	check_10day	check_1day	check_12hr
2016-06-29 15:45:47	FALSE	FALSE	FALSE	FALSE
2016-06-29 15:52:09	FALSE	FALSE	FALSE	FALSE
2016-06-29 17:22:24	FALSE	FALSE	FALSE	FALSE
2017-06-21 20:58:28	FALSE	FALSE	FALSE	FALSE
2017-06-26 15:33:21	FALSE	FALSE	FALSE	FALSE
2017-06-26 16:04:51	FALSE	FALSE	FALSE	FALSE
2017-06-26 16:11:11	FALSE	FALSE	FALSE	FALSE
2019-07-26 15:20:23	TRUE	TRUE	TRUE	TRUE
2019-07-26 16:22:11	TRUE	TRUE	TRUE	TRUE
2019-07-26 22:09:25	TRUE	TRUE	TRUE	TRUE
2019-07-27 15:27:04	TRUE	TRUE	TRUE	TRUE
2019-07-30 15:34:20	TRUE	TRUE	TRUE	TRUE
2020-06-24 17:36:27	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:26:15	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:18:44	TRUE	TRUE	TRUE	TRUE
2020-06-26 19:28:06	TRUE	TRUE	TRUE	TRUE
2020-07-02 17:14:10	TRUE	TRUE	TRUE	TRUE

flightline_datetime	check_30day	check_10day	check_1day	check_12hr
2019-07-26 15:20:23	TRUE	TRUE	TRUE	TRUE
2019-07-26 16:22:11	TRUE	TRUE	TRUE	TRUE
2019-07-26 22:09:25	TRUE	TRUE	TRUE	TRUE
2019-07-27 15:27:04	TRUE	TRUE	TRUE	TRUE
2019-07-30 15:34:20	TRUE	TRUE	TRUE	TRUE
2020-06-24 17:36:27	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:26:15	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:18:44	TRUE	TRUE	TRUE	TRUE
2020-06-26 19:28:06	TRUE	TRUE	TRUE	TRUE
2020-07-02 17:14:10	TRUE	TRUE	TRUE	TRUE

Helper table for manually pulling SUNA data (AOP dates):

Helper table for pulling L0 SUNA for flight dates with SUNA data.
Interval is 3 days before and after to facilitate downloads.
Connect to VPN and use DQ Blizzard. View Level 0 data, extract L0 sub-stream
stream 006 is Absorbance at 254
stream 007 is Absorbance at 350

flightline_datetime	check_1day	interval3_start	interval3_end
2019-07-26 15:20:23	TRUE	2019-07-23	2019-07-29
2019-07-26 16:22:11	TRUE	2019-07-23	2019-07-29
2019-07-26 22:09:25	TRUE	2019-07-23	2019-07-29
2019-07-27 15:27:04	TRUE	2019-07-24	2019-07-30
2019-07-30 15:34:20	TRUE	2019-07-27	2019-08-02
2020-06-24 17:36:27	TRUE	2020-06-21	2020-06-27
2020-06-24 16:26:15	TRUE	2020-06-21	2020-06-27
2020-06-24 16:18:44	TRUE	2020-06-21	2020-06-27
2020-06-26 19:28:06	TRUE	2020-06-23	2020-06-29
2020-07-02 17:14:10	TRUE	2020-06-29	2020-07-05

Those are the availabilities for the SUNA data at flight times.
Save 1 file for each year eg NEON-L0-SUNA/AOP-dates/{mysite}/{mysite}-103-2020.csv
Process raw suna data with process-L0-SUNA in processing project

2019 abs plots

## Rows: 803
## Columns: 9
## Delimiter: ","
## dbl  [8]: mean_abs254, sd_abs254, n_abs254, se_abs254, mean_abs350, sd_abs350, n_abs350,...
## dttm [1]: burst_id
## 
## Use `spec()` to retrieve the guessed column specification
## Pass a specification to the `col_types` argument to quiet this message

2019 abs avg

collectDateTime	datetime	mean_abs254	abs254_ma01	abs254_ma03	abs254_ma04	abs254_ma04u	abs254_ma06	abs254_ma12
2019-07-26 15:20:23	2019-07-26 15:15:00	0.7240	0.7223	0.7223	0.7224	0.7246	0.7224	0.7220
2019-07-26 16:22:11	2019-07-26 16:15:00	0.7213	0.7213	0.7220	0.7220	0.7235	0.7220	0.7218
2019-07-26 22:09:25	2019-07-26 22:15:00	0.7213	0.7201	0.7182	0.7182	0.7175	0.7182	0.7182
2019-07-27 15:27:04	2019-07-27 15:30:00	0.6946	0.6944	0.6928	0.6925	0.6904	0.6928	0.6931
2019-07-30 15:34:20	2019-07-30 15:30:00	0.6522	0.6524	0.6523	0.6531	0.6536	0.6560	0.6577

collectDateTime	datetime	mean_abs350	abs350_ma01	abs350_ma03	abs350_ma04	abs350_ma04u	abs350_ma06	abs350_ma12
2019-07-26 15:20:23	2019-07-26 15:15:00	0.1961	0.1945	0.1945	0.1945	0.1966	0.1945	0.1938
2019-07-26 16:22:11	2019-07-26 16:15:00	0.1936	0.1936	0.1936	0.1936	0.1954	0.1936	0.1931
2019-07-26 22:09:25	2019-07-26 22:15:00	0.1912	0.1905	0.1903	0.1903	0.1887	0.1903	0.1902
2019-07-27 15:27:04	2019-07-27 15:30:00	0.1660	0.1659	0.1654	0.1653	0.1636	0.1654	0.1655
2019-07-30 15:34:20	2019-07-30 15:30:00	0.1245	0.1245	0.1243	0.1258	0.1269	0.1278	0.1315

2020 abs plots

## Rows: 755
## Columns: 9
## Delimiter: ","
## dbl  [8]: mean_abs254, sd_abs254, n_abs254, se_abs254, mean_abs350, sd_abs350, n_abs350,...
## dttm [1]: burst_id
## 
## Use `spec()` to retrieve the guessed column specification
## Pass a specification to the `col_types` argument to quiet this message

2020 abs avg

collectDateTime	datetime	mean_abs254	abs254_ma01	abs254_ma03	abs254_ma04	abs254_ma04u	abs254_ma06	abs254_ma12
2020-06-24 17:36:27	2020-06-24 17:30:00	0.4602	0.4606	0.4606	0.4607	0.4609	0.4609	0.4611
2020-06-24 16:26:15	2020-06-24 16:30:00	0.4600	0.4600	0.4602	0.4604	0.4608	0.4609	0.4611
2020-06-24 16:18:44	2020-06-24 16:15:00	0.4598	0.4600	0.4602	0.4604	0.4609	0.4608	0.4612
2020-06-26 19:28:06	2020-06-26 19:30:00	0.4726	0.4727	0.4727	0.4727	0.4733	0.4727	0.4732

collectDateTime	datetime	mean_abs350	abs350_ma01	abs350_ma03	abs350_ma04	abs350_ma04u	abs350_ma06	abs350_ma12
2020-06-24 17:36:27	2020-06-24 17:30:00	0.0449	0.0450	0.0452	0.0452	0.0455	0.0457	0.0456
2020-06-24 16:26:15	2020-06-24 16:30:00	0.0449	0.0447	0.0448	0.0450	0.0453	0.0455	0.0457
2020-06-24 16:18:44	2020-06-24 16:15:00	0.0443	0.0447	0.0447	0.0450	0.0453	0.0454	0.0458
2020-06-26 19:28:06	2020-06-26 19:30:00	0.0545	0.0548	0.0548	0.0548	0.0550	0.0548	0.0554

x2019 SUNA plots 2019

filtered to no Quality Flags

x2020 SUNA plots 2020

Not filtered by QFs
QF because of lack of calibration?

II.(E) SUNA vs AOS absorbs

Use surface water chemistry data to see date and times of ground sample collection for UV absorbance data

This is the availability of SUNA data for AOS sampling dates

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2016-10-19 15:30:00	FALSE	FALSE	FALSE	FALSE
2016-11-16 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-12-19 17:53:00	FALSE	FALSE	FALSE	FALSE
2017-01-18 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-02-13 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-03-22 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-04-12 15:15:00	FALSE	FALSE	FALSE	FALSE
2017-05-03 14:20:00	FALSE	FALSE	FALSE	FALSE
2017-06-08 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-07-06 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-08-03 14:50:00	FALSE	FALSE	FALSE	FALSE
2017-09-06 14:15:00	FALSE	FALSE	FALSE	FALSE
2017-10-19 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-11-01 15:15:00	FALSE	FALSE	FALSE	FALSE
2018-01-04 17:00:00	FALSE	FALSE	FALSE	FALSE
2018-02-13 16:45:00	FALSE	FALSE	FALSE	FALSE
2018-03-13 16:05:00	FALSE	FALSE	FALSE	FALSE
2018-04-17 16:03:00	FALSE	FALSE	FALSE	FALSE
2018-05-01 16:26:00	FALSE	FALSE	FALSE	FALSE
2018-06-05 15:40:00	FALSE	FALSE	FALSE	FALSE
2018-07-05 15:20:00	FALSE	FALSE	FALSE	FALSE
2018-08-07 16:16:00	FALSE	FALSE	FALSE	FALSE
2018-09-04 15:50:00	FALSE	FALSE	FALSE	FALSE
2018-10-16 16:03:00	FALSE	FALSE	FALSE	FALSE
2018-10-30 16:07:00	FALSE	FALSE	FALSE	FALSE
2018-12-04 16:42:00	FALSE	FALSE	FALSE	FALSE
2019-01-10 17:10:00	FALSE	FALSE	FALSE	FALSE
2019-02-11 17:55:00	FALSE	FALSE	FALSE	FALSE
2019-03-26 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-05-07 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-06-04 15:25:00	FALSE	FALSE	FALSE	FALSE
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE
2019-12-04 17:40:00	FALSE	FALSE	FALSE	FALSE
2020-08-04 15:07:00	FALSE	FALSE	FALSE	FALSE
2020-09-02 13:27:00	FALSE	FALSE	FALSE	FALSE
2021-01-07 16:08:00	FALSE	FALSE	FALSE	FALSE
2021-02-02 17:15:00	FALSE	FALSE	FALSE	FALSE
2021-03-02 18:05:00	FALSE	FALSE	FALSE	FALSE
2021-04-07 15:45:00	TRUE	FALSE	FALSE	FALSE
2021-05-04 15:22:00	TRUE	TRUE	TRUE	TRUE

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE
2021-04-07 15:45:00	TRUE	FALSE	FALSE	FALSE
2021-05-04 15:22:00	TRUE	TRUE	TRUE	TRUE

Helper table for manually pulling SUNA data (AOS dates):

aos_datetime	check_1day	interval3_start	interval3_end
2019-07-01 14:53:00	TRUE	2019-06-28	2019-07-04
2019-08-06 15:55:00	TRUE	2019-08-03	2019-08-09
2019-09-04 16:00:00	TRUE	2019-09-01	2019-09-07
2019-10-16 17:05:00	TRUE	2019-10-13	2019-10-19
2021-04-07 15:45:00	FALSE	2021-04-04	2021-04-10
2021-05-04 15:22:00	TRUE	2021-05-01	2021-05-07

processed SUNA absorbance data are in box folder NEON-processed/suna-L0-timeseries
how do absorbances from nitrate sensor compare to AOS samples?

Regression for sensor data within the same day

## `geom_smooth()` using formula 'y ~ x'

III. Suspended Matter

III.(A) AOS Sampling

From processed Surface water chemistry files on Box
TDS passes through .45 micron filter
TSS is from weight of dried mass that doesnt pass through filter

Data from all locations at site

## geom_path: Each group consists of only one observation. Do you need to adjust
## the group aesthetic?
## geom_path: Each group consists of only one observation. Do you need to adjust
## the group aesthetic?
## geom_path: Each group consists of only one observation. Do you need to adjust
## the group aesthetic?

sm_site_df <- sm_site_df %>%
  dplyr::filter(namedLocation %in% my_loc)

## Warning: Transformation introduced infinite values in continuous y-axis

## Warning: Transformation introduced infinite values in continuous y-axis

How much does TDS/TSS change?

Adjusts sample IDs in a new column to group together raw and filtered, while keeping track of duplicates
Dissolved passes through .45 micron filter, suspended does not

## Warning: Removed 1 row(s) containing missing values (geom_path).

## Warning: Removed 2 rows containing missing values (geom_point).

Match sediment aos to aop dates

source('R/match-aop-aos.R')

sm_match_list <- match_aop_aos(mysite_dates, sm_site_df, 'collect_date')

knitr::kable(sm_match_list$dates)

flightdate	aos_before	aos_after	days_before	days_after	min_days	meets_thresh1	meets_thresh2
2016-06-29	2014-06-19	2016-10-19	741	112	112	FALSE	FALSE
2017-06-21	2017-06-08	2017-07-06	13	15	13	TRUE	TRUE
2017-06-26	2017-06-08	2017-07-06	18	10	10	TRUE	TRUE
2019-07-26	2019-07-01	2019-08-06	25	11	11	TRUE	TRUE
2019-07-27	2019-07-01	2019-08-06	26	10	10	TRUE	TRUE
2019-07-30	2019-07-01	2019-08-06	29	7	7	TRUE	TRUE
2020-06-24	2019-12-04	2021-05-04	203	314	203	FALSE	FALSE
2020-06-26	2019-12-04	2021-05-04	205	312	205	FALSE	FALSE
2020-07-02	2019-12-04	2021-05-04	211	306	211	FALSE	FALSE

knitr::kable(sm_match_list$matches)

flightdate	aos_match	days
2017-06-21	2017-06-08	13
2017-06-26	2017-07-06	10
2019-07-26	2019-08-06	11
2019-07-27	2019-08-06	10
2019-07-30	2019-08-06	7

TSS x flightdate

These are the closest in time ground-based measurements of TDS (mg/L) and TSS (mg/L)

List of values means there are duplicate samples (multiple sampleIDs) from same date.

aos_match	flightdate	days	TDS	TSS	TSS - Dry Mass
2017-06-08	2017-06-21	13	910.1	32	800
2017-07-06	2017-06-26	10	885.62, 887.64, 891.23	10, 9, 9	900, 900, 1000
2019-08-06	2019-07-26	11	744.6	10	500
2019-08-06	2019-07-27	10	744.6	10	500
2019-08-06	2019-07-30	7	744.6	10	500

III.(B) Turbidity AIS time series

The EXO turbidity sensor employs a near‐IR light source (~780 ‐ 900 nm) and detects scattering at 90 degrees of the incident beam.

processed turbidity water quality time series on box

flightline_datetime	check_30day	check_10day	check_1day	check_12hr
2016-06-29 15:45:47	FALSE	FALSE	FALSE	FALSE
2016-06-29 15:52:09	FALSE	FALSE	FALSE	FALSE
2016-06-29 17:22:24	FALSE	FALSE	FALSE	FALSE
2017-06-21 20:58:28	FALSE	FALSE	FALSE	FALSE
2017-06-26 15:33:21	FALSE	FALSE	FALSE	FALSE
2017-06-26 16:04:51	FALSE	FALSE	FALSE	FALSE
2017-06-26 16:11:11	FALSE	FALSE	FALSE	FALSE
2019-07-26 15:20:23	TRUE	TRUE	TRUE	TRUE
2019-07-26 16:22:11	TRUE	TRUE	TRUE	TRUE
2019-07-26 22:09:25	TRUE	TRUE	TRUE	TRUE
2019-07-27 15:27:04	TRUE	TRUE	TRUE	TRUE
2019-07-30 15:34:20	TRUE	TRUE	TRUE	TRUE
2020-06-24 17:36:27	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:26:15	TRUE	TRUE	TRUE	TRUE
2020-06-24 16:18:44	TRUE	TRUE	TRUE	TRUE
2020-06-26 19:28:06	TRUE	TRUE	TRUE	FALSE
2020-07-02 17:14:10	TRUE	TRUE	FALSE	FALSE

2019 Turbidity plots

filtered to no quality flags

2020 Turbidity plots

Not filtered for quality flags
QF because of lack of calibration?

## Warning: Removed 8537 row(s) containing missing values (geom_path).

## Warning: Removed 8658 rows containing missing values (geom_point).

## Warning: Removed 2438 row(s) containing missing values (geom_path).

## Warning: Removed 2552 rows containing missing values (geom_point).

## Warning: Removed 444 row(s) containing missing values (geom_path).

## Warning: Removed 444 rows containing missing values (geom_point).

## Warning: Removed 4737 row(s) containing missing values (geom_path).

## Warning: Removed 4737 rows containing missing values (geom_point).

2019 turb avg

Smooth the fdom time series using a rolling mean with zoo::rollapply.
First convert to a regular 5 minute time series using tsibble

Get the moving average value closet to flight time.

5min is the original data from that row
ma01 = 1 hour window, ma03 = 3 hour window, and so on
chl_ma04u is mean, the others are all mean

collectDateTime	datetime	turb5min	ma01	ma03	ma04	ma04u	ma06	ma12
2019-07-26 15:20:23	2019-07-26 15:20:00	2.09	2.170	2.170	2.175	2.228913	2.180	2.240
2019-07-26 16:22:11	2019-07-26 16:20:00	2.17	2.205	2.230	2.240	2.326667	2.245	2.290
2019-07-26 22:09:25	2019-07-26 22:10:00	3.98	3.185	3.140	3.125	3.129167	3.080	2.925
2019-07-27 15:27:04	2019-07-27 15:25:00	3.62	2.985	2.930	2.910	2.942917	2.790	2.600
2019-07-30 15:34:20	2019-07-30 15:35:00	2.75	2.770	2.795	2.820	2.921591	2.820	2.820

2020 turb avg

Smooth the fdom time series using a rolling mean with zoo::rollapply.
First convert to a regular 5 minute time series using tsibble

Get the moving average value closet to flight time.

5min is the original data from that row
ma01 = 1 hour window, ma03 = 3 hour window, and so on
chl_ma04u is mean, the others are all mean

collectDateTime	datetime	turb5min	ma01	ma03	ma04	ma04u	ma06	ma12
2020-06-24 17:36:27	2020-06-24 17:35:00	1.91	1.865	1.820	1.865	3.284375	1.845	1.9
2020-06-24 16:26:15	2020-06-24 16:25:00	1.51	1.795	1.875	1.840	2.687708	1.875	1.9
2020-06-24 16:18:44	2020-06-24 16:20:00	1.93	1.795	1.875	1.815	2.677292	1.875	1.9
2020-06-26 19:28:06	2020-06-26 19:30:00	NA	NA	NA	NA	NaN	NA	NA
2020-07-02 17:14:10	2020-07-02 17:15:00	NA	NA	NA	NA	NaN	NA	NA

III.(C) Turbidity vs TDS/ TSS (TODO)

use swchem_df date and times with the aop/ais matching function to determine which sampling dates have sensor data from same day around time of sampling.
Get fDOM sensor data from all AOS sampling dates (use ais with buffer2 function?)
show what is available
calculate rolling averages of fDOM time series
get rolling average values for time closest to sampling using which min on difftime
little overlap because of seasonal removal of buoys

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2014-05-14 15:15:00	FALSE	FALSE	FALSE	FALSE
2014-06-19 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-10-19 15:30:00	FALSE	FALSE	FALSE	FALSE
2016-11-16 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-12-19 17:53:00	FALSE	FALSE	FALSE	FALSE
2017-01-18 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-02-13 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-03-22 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-04-12 15:15:00	FALSE	FALSE	FALSE	FALSE
2017-05-03 14:20:00	FALSE	FALSE	FALSE	FALSE
2017-06-08 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-07-06 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-08-03 14:50:00	FALSE	FALSE	FALSE	FALSE
2017-09-06 14:15:00	FALSE	FALSE	FALSE	FALSE
2017-10-19 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-11-01 15:15:00	FALSE	FALSE	FALSE	FALSE
2018-01-04 17:00:00	FALSE	FALSE	FALSE	FALSE
2018-02-13 16:45:00	FALSE	FALSE	FALSE	FALSE
2018-03-13 16:05:00	FALSE	FALSE	FALSE	FALSE
2018-04-17 16:03:00	FALSE	FALSE	FALSE	FALSE
2018-05-01 16:26:00	FALSE	FALSE	FALSE	FALSE
2018-06-05 15:40:00	FALSE	FALSE	FALSE	FALSE
2018-07-05 15:20:00	TRUE	TRUE	TRUE	TRUE
2018-08-07 16:16:00	TRUE	TRUE	TRUE	TRUE
2018-09-04 15:50:00	TRUE	TRUE	TRUE	TRUE
2018-10-16 16:03:00	TRUE	TRUE	TRUE	TRUE
2018-10-30 16:07:00	FALSE	FALSE	FALSE	FALSE
2018-12-04 16:42:00	FALSE	FALSE	FALSE	FALSE
2019-01-10 17:10:00	FALSE	FALSE	FALSE	FALSE
2019-02-11 17:55:00	FALSE	FALSE	FALSE	FALSE
2019-03-26 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-05-07 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-06-04 15:25:00	TRUE	TRUE	TRUE	TRUE
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE
2019-12-04 17:40:00	FALSE	FALSE	FALSE	FALSE
2021-05-04 15:22:00	FALSE	FALSE	FALSE	FALSE

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2018-07-05 15:20:00	TRUE	TRUE	TRUE	TRUE
2018-08-07 16:16:00	TRUE	TRUE	TRUE	TRUE
2018-09-04 15:50:00	TRUE	TRUE	TRUE	TRUE
2018-10-16 16:03:00	TRUE	TRUE	TRUE	TRUE
2019-06-04 15:25:00	TRUE	TRUE	TRUE	TRUE
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE

collect_date	TSS	TDS
2018-07-05	4	945.20
2018-08-07	6	935.64
2018-09-04	10	977.41
2018-10-16	34	1053.00
2019-06-04	8	872.15
2019-07-01	6	921.41
2019-07-01	8	915.63
2019-07-01	6	918.42
2019-08-06	10	744.60
2019-09-04	20	950.23
2019-09-04	18	942.21
2019-09-04	20	958.42
2019-10-16	12	833.27

Same plot as above with only QF = 0 data

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2014-05-14 15:15:00	FALSE	FALSE	FALSE	FALSE
2014-06-19 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-10-19 15:30:00	FALSE	FALSE	FALSE	FALSE
2016-11-16 16:00:00	FALSE	FALSE	FALSE	FALSE
2016-12-19 17:53:00	FALSE	FALSE	FALSE	FALSE
2017-01-18 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-02-13 16:15:00	FALSE	FALSE	FALSE	FALSE
2017-03-22 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-04-12 15:15:00	FALSE	FALSE	FALSE	FALSE
2017-05-03 14:20:00	FALSE	FALSE	FALSE	FALSE
2017-06-08 14:45:00	FALSE	FALSE	FALSE	FALSE
2017-07-06 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-08-03 14:50:00	FALSE	FALSE	FALSE	FALSE
2017-09-06 14:15:00	FALSE	FALSE	FALSE	FALSE
2017-10-19 15:00:00	FALSE	FALSE	FALSE	FALSE
2017-11-01 15:15:00	FALSE	FALSE	FALSE	FALSE
2018-01-04 17:00:00	FALSE	FALSE	FALSE	FALSE
2018-02-13 16:45:00	FALSE	FALSE	FALSE	FALSE
2018-03-13 16:05:00	FALSE	FALSE	FALSE	FALSE
2018-04-17 16:03:00	FALSE	FALSE	FALSE	FALSE
2018-05-01 16:26:00	FALSE	FALSE	FALSE	FALSE
2018-06-05 15:40:00	FALSE	FALSE	FALSE	FALSE
2018-07-05 15:20:00	TRUE	TRUE	TRUE	TRUE
2018-08-07 16:16:00	TRUE	TRUE	TRUE	TRUE
2018-09-04 15:50:00	TRUE	TRUE	TRUE	TRUE
2018-10-16 16:03:00	TRUE	TRUE	TRUE	TRUE
2018-10-30 16:07:00	FALSE	FALSE	FALSE	FALSE
2018-12-04 16:42:00	FALSE	FALSE	FALSE	FALSE
2019-01-10 17:10:00	FALSE	FALSE	FALSE	FALSE
2019-02-11 17:55:00	FALSE	FALSE	FALSE	FALSE
2019-03-26 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-05-07 16:10:00	FALSE	FALSE	FALSE	FALSE
2019-06-04 15:25:00	TRUE	TRUE	TRUE	TRUE
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE
2019-12-04 17:40:00	FALSE	FALSE	FALSE	FALSE
2021-05-04 15:22:00	FALSE	FALSE	FALSE	FALSE

aos_datetime	check_3day	check_1day	check_6hr	check_1hr
2018-07-05 15:20:00	TRUE	TRUE	TRUE	TRUE
2018-08-07 16:16:00	TRUE	TRUE	TRUE	TRUE
2018-09-04 15:50:00	TRUE	TRUE	TRUE	TRUE
2018-10-16 16:03:00	TRUE	TRUE	TRUE	TRUE
2019-06-04 15:25:00	TRUE	TRUE	TRUE	TRUE
2019-07-01 14:53:00	TRUE	TRUE	TRUE	TRUE
2019-08-06 15:55:00	TRUE	TRUE	TRUE	TRUE
2019-09-04 16:00:00	TRUE	TRUE	TRUE	TRUE
2019-10-16 17:05:00	TRUE	TRUE	TRUE	TRUE

## Warning: Removed 1279 rows containing missing values (geom_point).

Compare AOS and AIS

Colors indicate month

Ground based data

Summary of data overlap for PRLA

I. Chlorophyll

I.(A) AOS Sampling

chl AOS x flightdates

Algal Community

I.(B) Chl AIS time series

2019 chl AIS plots

2020 chl AIS plots

2019 Chl avg

2020 Chl avg

I.(C) Chl AIS vs AOS

II. DOM

II.(A) AOS Sampling

DOC x flightdates

UV abs x flightdates

TOC x flightdates

Fe x flightdates

II.(B) fDOM AIS time series

2019 fDOM plots

2020 fDOM plots

2019 fDOM avg

2020 fDOM avg

II.(C) fDOM vs AOS DOM

II.(D) SUNA AIS time series

Helper table for manually pulling SUNA data (AOP dates):

2019 abs plots

2019 abs avg

2020 abs plots

2020 abs avg

x2019 SUNA plots 2019

x2020 SUNA plots 2020

II.(E) SUNA vs AOS absorbs

III. Suspended Matter

III.(A) AOS Sampling

TSS x flightdate

III.(B) Turbidity AIS time series

2019 Turbidity plots

2020 Turbidity plots

2019 turb avg

2020 turb avg

III.(C) Turbidity vs TDS/ TSS (TODO)